Added top n log probs #2262

shanmugamr1992 · 2025-11-15T00:01:16Z

What does this PR do ?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share discuss a design-doc with the team.

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either [email protected] or [email protected].

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

copy-pr-bot · 2025-11-15T00:01:19Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

megatron/core/inference/text_generation_controllers/text_generation_controller.py

…M into dynamic_latest

kvareddy · 2025-11-19T17:49:48Z

@santhnm2 can you please take a look at this MR?

megatron/core/inference/contexts/dynamic_context.py

megatron/core/inference/engines/dynamic_engine.py

examples/inference/gpt/gpt_dynamic_inference.py

megatron/core/inference/text_generation_controllers/text_generation_controller.py

tdene · 2025-11-22T18:12:07Z

I left a few nit comments, but the general code looks good to me.

I will approve this after main is merged into it.

shanmugamr1992 · 2025-11-25T17:43:46Z

/ok

shanmugamr1992 · 2025-11-25T20:15:11Z

/ok to test 300c130

shanmugamr1992 · 2025-11-25T21:09:46Z

/ok to test 4303dd0

shanmugamr1992 · 2025-11-25T21:32:38Z

/ok to test 0740fd6

Added top n log probs

2cbc985

shanmugamr1992 requested a review from a team as a code owner November 15, 2025 00:01

kvareddy requested a review from tdene November 15, 2025 06:52

Merge branch 'main' into dynamic_latest

724f80b

ko3n1g requested review from a team as code owners November 15, 2025 17:09

tdene reviewed Nov 17, 2025

View reviewed changes

megatron/core/inference/text_generation_controllers/text_generation_controller.py Outdated Show resolved Hide resolved

tdene reviewed Nov 17, 2025

View reviewed changes

megatron/core/inference/text_generation_controllers/text_generation_controller.py Outdated Show resolved Hide resolved

tdene reviewed Nov 17, 2025

View reviewed changes

megatron/core/inference/text_generation_controllers/text_generation_controller.py Outdated Show resolved Hide resolved

tdene mentioned this pull request Nov 17, 2025

Top n dynamic #2260

Closed

6 tasks

Shanmugam Ramasamy added 4 commits November 17, 2025 18:01

Added prompt log probs

d6286fd

Merge branch 'dynamic_latest' of github.com:shanmugamr1992/Megatron-L…

14247a5

…M into dynamic_latest

Added prompt log probs

bc4981c

Added prompt log probs

d22a9b7

Shanmugam Ramasamy added 2 commits November 19, 2025 10:55

Added prompt log probs

a8e4719

Added prompt log probs

4d3ebf1

santhnm2 approved these changes Nov 19, 2025

View reviewed changes

megatron/core/inference/contexts/dynamic_context.py Show resolved Hide resolved

megatron/core/inference/engines/dynamic_engine.py Outdated Show resolved Hide resolved

Shanmugam Ramasamy added 2 commits November 19, 2025 14:41

Added prompt log probs

4a7e96e

Added prompt log probs

af4cb7a

tdene reviewed Nov 22, 2025

View reviewed changes

examples/inference/gpt/gpt_dynamic_inference.py Outdated Show resolved Hide resolved

tdene reviewed Nov 22, 2025

View reviewed changes

examples/inference/gpt/gpt_dynamic_inference.py Show resolved Hide resolved

tdene reviewed Nov 22, 2025

View reviewed changes

examples/inference/gpt/gpt_dynamic_inference.py Show resolved Hide resolved

tdene reviewed Nov 22, 2025

View reviewed changes

megatron/core/inference/text_generation_controllers/text_generation_controller.py Outdated Show resolved Hide resolved

tdene reviewed Nov 22, 2025

View reviewed changes

megatron/core/inference/text_generation_controllers/text_generation_controller.py Outdated Show resolved Hide resolved

Shanmugam Ramasamy added 2 commits November 24, 2025 14:44

Added prompt log probs

144e017

Added prompt log probs

6a0935b

tdene approved these changes Nov 24, 2025

View reviewed changes

kvareddy approved these changes Nov 25, 2025

View reviewed changes

shanmugamr1992 added the Run tests label Nov 25, 2025

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 20:15 Inactive

ko3n1g added this to the Core 0.16 milestone Nov 25, 2025

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 20:15 Inactive

copy-pr-bot bot temporarily deployed to test November 25, 2025 20:16 Inactive

shanmugamr1992 and others added 2 commits November 25, 2025 13:07

Bug fix

16fa838

Merge branch 'main' into dynamic_latest

4303dd0

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 21:09 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 21:10 Inactive

copy-pr-bot bot temporarily deployed to test November 25, 2025 21:10 Inactive

Addressed comments and fixed issue

0740fd6

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 21:32 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 25, 2025 21:33 Inactive

copy-pr-bot bot temporarily deployed to test November 25, 2025 21:33 Inactive

shanmugamr1992 enabled auto-merge November 25, 2025 21:52

shanmugamr1992 added this pull request to the merge queue Nov 25, 2025

Merged via the queue into NVIDIA:main with commit 898d633 Nov 25, 2025
127 checks passed

shanmugamr1992 deleted the dynamic_latest branch November 25, 2025 22:22

Added top n log probs #2262

Added top n log probs #2262

Uh oh!

Conversation

shanmugamr1992 commented Nov 15, 2025

What does this PR do ?

Contribution process

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot bot commented Nov 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kvareddy commented Nov 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdene commented Nov 22, 2025

Uh oh!

shanmugamr1992 commented Nov 25, 2025

Uh oh!

shanmugamr1992 commented Nov 25, 2025

Uh oh!

shanmugamr1992 commented Nov 25, 2025

Uh oh!

shanmugamr1992 commented Nov 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

(Step 1): Add PR label `Expert Review`